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1.0  INTRODUCTION 


vGADS  is  the  Graphic  Attitude  Determination  System  currently  in  use  at  Eglin  Air  Force 
Base  for  manuedly  “reducing  stores  separation  data  from  16mm  film.”  This  process  involves 
the  m£inual  matching  of  a  wireframe  model  to  the  projected  image  of  the  weapon  for  each 
frame  of  the  film.  Using  the  information  from  the  matched  wireframe  model,  the  camera 
location  and  opticail  characteristics,  it  is  possible  to  determine  the  position  of  the  store  as  a 
function  of  time.  While  the  technique  has  been  very  successful,  it  is  labor  intensive  and  a 
fatiguing  process  for  the  operator.  Manual  operation  also  raises  the  chaillenge  of  consistent 
matching  within  a  film  sequence  and  between  operators.  Finally,  a  quantitative  criteria  for 
determining  model  position  cannot  be  assessed  with  the  present  manual  system.  As  such, 
assistance  for  the  operator  in  the  form  of  additional  information,  options  on  how  information 
is  presented,  and  ideally  an  automatic  matching  capability  is  highly  desirable.  This  paper 
outlines  upgrades  to  the  manual  operation  at  the  algorithmic  level,  focussing  on  the  fecisibility 
of  automatically  matching  model  data  to  images  from  film.  -  ‘ 


2.0  LITERATURE  REVIEW 

An  extensive  literature  search  was  undertaken  at  the  beginning  of  this  project  to  assess 
the  applicability  of  recent  research  to  the  problems  posed  by  automated  GADS.  The  details 
of  this  survey  are  included  in  Appendix  A.- From  the  several  hundred  titles  of  potential  rele¬ 
vance,  over  20  articles  were  requested  and  reviewed.  While  some  of  the  techniques  discussed 
could  be  beneficial  in  portions  of  a  final  automated  algorithm,  most  were  extremely  specific 
and  restrictive  in  the  problem  addressed  and  the  solution  tendered. 

In  the  GADS  environment,  model  matching  occurs  on  noisy  images  with  obscured  mov¬ 
ing  targets  taken  from  a  nonstationary  camera  under  highly  variable  lighting  conditions  and 
target  backgrounds.  This  scenario  provides  a  rich  source  of  topics,  but  few  of  the  papers 
examined  dealt  with  more  than  one  or  two  of  the  simultaneous  concert  of  conditions  in  GADS. 
The  previous  research  of  greatest  benefit  was  discovered  at  the  IEEE  Sixteenth  Workshop 
on  Applied  Imagery  Pattern  Recognition  held  October  28-30,  1987,  in  Washington,  D.C., 
attended  by  ERIM  representatives  including  this  project’s  program  manager.  Dr.  James  Sher¬ 
man.  A.  S.  Politopoulos  presented  a  paper  on  ATR  recognition  with  a  technique  for  corre¬ 
lation  of  2  line  models  using  a  minimum  distance  metric.  From  this  kernel  idea,  the  use  of 
"potential  fields”  was  developed  and  incorporated  in  the  design  of  an  automatic  algorithm. 


3.0  ALGORITHM  OVERVIEW 

ERIM  advocates  an  evolutionary,  incremental  approach  to  developing  an  automated  GADS 
capability,  building  on  the  experience  gained  from  manual  operations  and  utilizing  exist¬ 
ing  hardware  and  techniques  as  appropriate.  From  an  operator's  viewpoint,  an  automated 


1 


GADS  system  would  be  the  Scime  as  the  manual  GADS  system  with  the  addition  of  some 
menu  options  to  provide  automatic  matching  of  the  wireframe*  model  to  the  image  data.  In 
this  approach,  the  operator  has  the  capability  of  switching  between  automatic  and  manual 
matching  at  any  time.  Information  similcu:  to  that  utilized  for  automatic  model  placement 
would  also  be  optionally  available  to  aid  the  operator  in  manual  mode  and  in  quamtitativelv 
assessing  tlie  model  fit  to  the  image.  The  basic  sequence  for  automatic  mode  is  as  follows: 

(1)  .An  operator  initializes  the  system  by  selecting  the  first  frame,  appropriate  model,  and 
positioning  the  model  “close  enough”  to  the  target  before  selecting  automatic  process¬ 
ing. 

(2)  The  algorithm  extracts  features  from  the  image  with  correspondence  to  features  in  target 
model,  e.g.,  for  wireframe  models,  edges  would  be  extracted  from  the  image. 

(3)  A  “potential  field”  is  generated  around  the  target  model. 

Note  that  the  location  would  be  the  operator  positioned  model  for  initialization,  the 
previous  frame’s  final  model  position  for  the  start  of  the  next  frame,  (optionally  modified 
by  autoprediction),  or  a  candidate  model  position  when  iterating  for  a  best  fit. 

(4)  The  e.xtracted  image  features  are  placed  in  the  model’s  “potential  field”  and  the  “gravi¬ 
tational  forces”  acting  on  the  features  axe  calculated.  Topographically,  the  model  forms 
a  valley  with  mountains  of  constant  slope  on  all  sides.  When  the  image  features  are 
placed  in  this  terrain,  they  will  slide  down  towards  the  valley  until  the  lowest  possible 
position  is  reached. 

(•5)  The  information  obtained  from  step-4  is  used  to  shift  the  target  model  so  as  to  minimize 
the  “forces”  acting  on  the  image  features  from  the  model  “potential  field.”  Because  this 
is  an  approximate  positioning,  the  algorithm  loops  back  to  step  3  and  iterates  until  an 
overall  minimum  is  achieved. 

(6)  The  algorithm  outputs  the  results  of  the  model  fit,  including  a  confidence  measure  for 
each  of  the  6  degrees  of  freedom  reported. 

(7)  Finally,  the  algorithm  determines  the  next  frame  in  the  sequence  (if  any),  digitizes  the 
image,  and  repeats  the  matching  process  starting  at  step  2.  Note  that  the  next  frame 
may  be  forward  or  backward  in  time,  may  be  more  than  one  frame  distant,  or  may  be 
selected  by  the  operator. 

Throughout  this  sequence,  the  operator  retains  the  option  of  extensive  review  and  veto  pow¬ 
ers  over  the  model  placement.  It  will  be  possible  to  interrupt  automatic  model  positioning, 
reposition  the  model  manually,  and  restart  automatic  mode  from  the  manual  placement. 
The  initial  model  position  being  “close  enough”  refers  to  placement  that  leads  to  closure  on 
the  best  fit.  Currently,  it’s  unlikely  the  algorithm  will  close  to  the  proper  model  fit  given 
an  initial  rotation  of  45°  or  more  from  truth  for  yaw  and  pitch  and  22°  for  roll.  While  it's 
anticipated  that  this  restriction  will  be  minor,  “close  enough”  is  a  function  of  the  model,  its 
orientation,  and  the  imagery  being  processed.  .As  such,  no  single  number  bounding  “close 
enough’  is  possible  without  restricting  other  system  parameters.  Consider  the  extreme  case 
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of  a  bomb  largely  obscured  by  the  aircraft  with  the  exposed  portion  completely  obscured  (for 
a  single  frame)  by  vapor.  In  this  instance  “close  enough”  doesn’t  exist,  and  the  algorithm 
would  firmly  request  the  operator  to  select  a  different  frame  for  initialization. 


4.0  CRITICAL  ALGORITHM  COMPONENTS 

In  the  development  of  a  new  algorithm,  a  scale  of  feasibility  risk  will  evolve  for  vairious 
portions  of  the  approach.  To  assure  the  overall  vadidity  of  am  approach,  the  highest  risk  sec¬ 
tions  of  an  algorithm  must  be  selected  and  developed.  In  this  instance,  the  precise  matching 
of  image  features  to  model  data  was  of  greatest  concern  to  ERIM  staff.  Accordingly,  steps 
3-5  were  identified  as  those  of  a  critical  nature  and  were  developed  aind  demonstrated  to 
Eglin  personnel.  Of  central  importance  to  the  matching  technique  was  the  development  of 
an  appropriate  field  of  values  axound  the  model  data.  The  fields  used  cire  fully  discussed 
in  the  Model  Field  Potential  section.  Technical  details  on  how  matching  is  accomplished 
are  given  in  the  .Algorithm  Calculation  section.  ERIM’s  focus  on  feature  matching  to  model 
data  should  not  be  construed  as  trivializing  the  other  steps  in  the  algorithm.  In  particular, 
extraction  of  features  from  the  image,  step  2,  represents  a  significant  challeuge.  From  the 
results  of  these  efforts,  ERIM  is  confident  of  the  overall  feasibility  of  an  automated  GADS 
system. 


5.0  MODEL  POTENTIAL  FIELD 

In  the  automated  GADS  environment,  exact  correspondence  between  model  pixels  and 
image  pixels  is  unknown,  and  initial  model  placement  in  the  image  data  is  only  approxi¬ 
mate.  The  feature  data  extracted  from  the  image  can  be  expected  to  contain  noise,  and  not 
to  contain  features  in  desired  locations  due  to  a  variety  of  factors  including  obscuration  and 
lighting  variations.  .As  such,  a  correlation  technique  which  is  relatively  insensitive  to  these 
factors  needed  to  be  developed.  Drawing  upon  the  analogy  of  gravitational  or  electromag¬ 
netic  fields,  ERIM  developed  a  potential  field  which  gives  the  distance  to  the  nearest  pixel  on 
the  model.  That  is,  the  field  answers  the  question  of  how  many  lines  and  columns  away  is  a 
model  pixel  as  a  function  of  location  in  the  image. 

The  pixel  neighborhood  operations  used  to  generate  the  fields  are  shown  in  Figure  1. 
The  distance  field  consisting  of  both  horizontal  (dx)  and  vertical  (dy)  components  gives  the 
total  number  of  x  and  y  shifts  to  the  closest  model  pixel.  This  scalar  field,  or  composite 
potential  field,  is  shown  in  the  perspective  plot  of  Figure  2,  where  the  “model”  source  is 
simply  two  intersecting  line  segments.  The  dx  field  calculation  shown  in  Figure  1  gives  the 
magnitude  of  the  horizontal  component  of  the  potential  field.  Similarly,  the  vertical  field 
calculations  are  given  with  the  dy  kernal  of  Figure  1.  These  component  fields  are  used  in 
the  feasibility  algorithm  to  independently  determine  the  line  and  column  shift.  In  other 
words,  the  horizontal  field  gives  the  distance  to  the  nearest  model  pixel  in  x.  Concomitantly, 


3 


the  verticaii  field  gives  the  line  distcince  for  the  minimum  shift  to  a  model  pixel.  Use  of 
component  fields  helped  simplify  the  algorithm  ctdculations,  but  required  more  effort  to 
develop  an  intuition  concerning  their  correctness  than  the  familiar  topology  of  a  potential 
field.  For  example,  is  the  vertical  band  of  zeroes  in  the  dx  field  of  Figure  1  correct?  It 
would  appecir  all  the  dx  field  veilues  originate  solely  from  the  endpoints  of  the  model.  In 
this  instance,  the  band  of  zeroes  indicate  that  no  shift  in  x  is  required  to  reaxih  the  nearest 
model  pixel,  i.e.,  the  nearest  model  pixel  can  be  reached  with  only  vertical  shifts.  The 
horizontal  and  vertical  components  of  the  potential  field  in  Figure  2  are  shown  in  Figures  3 
and  4.  Again,  these  component  fields  do  not  correspond  to  intuition  given  the  topology  of 
the  potential  field  in  Figure  2. 

When  working  with  these  prototype  fields,  an  interesting  situation  arises.  What  happens 
when  the  total  shift,  as  given  by  the  potential  field,  gives  more  than  one  path  to  a  model 
pixel?  Consider  a  45®  line  and  a  field  pixel  diagonally  adjacent  to  this  line.  In  such  a  case, 
the  total  number  of  shifts  needed  to  reach  the  source  line  is  two.  Unfortunately,  any  one  of 
three  shift  combinations  give  a  path  to  the  line:  2  shifts  in  y,  1  shift  in  y  plus  1  in  x,  or  2 
shifts  in  x.  The  prototype  fields  attempt  to  address  this  situation  by  giving  the  unambiguous 
portion  of  the  shift.  This  results  in  pixels  being  moved  to  locations  where  they  are  equally 
unresistant  to  shifts  in  either  direction.  Our  presumption  is  that  pixels  with  unambiguous 
paths  would  provide  the  final  push  to  align  model  and  feature  data.  Anyone  with  an  analytic 
mind  should  now  be  smiling  at  the  thought  of  pathologic  cases.  Careful  examination  of  the 
component  fields  in  Figures  3  and  4  compared  with  the  potential  field  of  Figure  2  will  reveal 
such  a  pathologic  case.  Notice  the  area  inside  the  model  lines  has  no  horizontal  or  vertical 
field  component.  This  arises  from  the  fact  that  any  pixel  inside  that  region  can  reach  a  model 
pixel  by  more  than  one  equidistant  combination  of  shifts.  While  these  prototype  fields  were 
sufficient  for  assessing  feasibility,  it’s  apparent  that  further  work  will  be  needed  to  satisfy 
the  performance  requirements  in  an  automated  GADS  environment. 


6.0  EXAMPLE  SEQUENCE 

Figures  5  through  17  provide  a  pictoriaJ  example  for  most  of  the  steps  in  the  algorithm 
overview.  Figure  5  simply  provides  an  example  frame  from  a  filmed  drop  as  might  be  selected 
by  an  operator  during  initialization.  Note  that  this  image  was  not  acquired  by  the  equipment 
used  in  manual  GADS.  Images  from  manual  GADS  will  provide  improved  quality.  Step  2  in  the 
algorithm  is  dramatized  in  Figures  6  through  8,  in  which  features  are  extracted  and  refined. 
The  'idealized  feature  extraction”  in  Figure  8  used  model  generated  data.  Use  of  model 
generated  data  for  feature  matching  enabled  ERIM  staff  to  focus  on  the  correlation  problem 
and  provided  an  unequivocal  criteria  for  best  fit,  i.e.,  the  two  models  should  ultimately  match 
perfectly. 

Figure  9  gives  a  hypothetical  start  condition,  which  is  a'‘close  enough  ’  placement  of  the 
model  (shown  in  green)  relative  to  the  image  features  (shown  in  red).  This  example  uses 
four  of  six  degrees  of  freedom  where  the  start  position  reflects  a  displacement  in  x,  y.  z,  and 
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pitch.  The  out-of-plaxie  tuigles,  in  this  case  yaw  cind  roll,  were  not  disturbed  nor  were  they 
permitted  to  change  in  the  iterations.  Figures  10  and  11  show  the  horizontal  and  vertical 
components  of  the  potenticd  field  corresponding  to  step  3  in  the  algorithm  overview.  The 
use  of  color  in  the  fields  is  simply  for  visucil  aid  in  determining  distaince.  Small  distances 
are  colored  red,  furthest  are  colored  violet,  and  the  isobar  bands  of  blue  are  given  every  10 
pixels. 

.Algorithm  step  4  is  shown  in  Figures  12  md  13  where  the  image  features  are  placed 
in  the  model’s  potential  field  and  analyzed.  For  the  horizontal  field,  171  pixels  require  an 
addition  to  their  column  location,  i.e.,  a  shift  right,  while  227  pixels  are  voting  for  a  left  shift 
and  332  have  a  zero  distance  shift.  Close  examination  of  Figure  13  shows  20  pixels  voting 
to  shift  down,  while  mother  31  extend  beyond  the  field.  Both  of  these  conditions  represent 
violations  of  "close  enough”  model  placement  for  the  current  algorithm.  Counting  the  blue 
isobar  lines  reveals  that  the  20  green  feature  pixels  in  Figure  13  are  greater  than  127  pixels 
distant  from  the  model.  Since  shifts  me  stored  as  signed  8-bit  quantities,  this  results  in  the 
20  pixels  changing  sign  and  voting  for  a  large  shift  in  the  wrong  direction.  In  this  instance, 
the  cumulative  voting  of  the  image  feature  pixels  was  still  able  to  provide  a  stable  response. 

The  results  of  the  first  iteration  are  shown  in  Figure  14  and  correspond  to  step  5  in  the 
algorithm  overview.  The  model  is  shifted  using  a  least  squares  calculation  from  the  image 
feature  data  placed  in  the  potential  field.  Considering  the  correct  correlation  is  unknown, 
model  placement  is  only  approximate  and  the  process  must  be  repeated. 

Figures  15  through  17  give  the  results  for  the  20‘*  iteration.  Figure  17  contains  the 
history  of  model  placement  in  each  iteration  and  reveals  how  rapidly  the  model  closes  upon 
image  features.  Finally.  Figures  18  through  21  give  a  graphical  analysis  of  model  placement 
relative  to  feature  location. 


7.0  ALGORITHM  INPUTS 

An  automated  GADS  capability  will  require  several  inputs,  the  majority  of  which  are 
identical  to  those  of  manual  GADS.  The  main  inputs  of  interest  for  both  systems  are  listed 
below: 

•  Target  model  data 

•  Digitized  imagery 

•  Operator  requests 

.Automated  GADS  will  input  model  data  by  calling  the  program  for  generating  wireframe 
models  in  the  manual  GADS  operation.  Utilization  of  existing  software  and  techniques  has 
an  obvious  cost  benefit,  assuming  the  following  conditions  are  met: 

•  .An  interface  between  the  model  generated  data  and  an  automated  GADS  system  ran  be 
accomplished  with  minimal  effort.  ERI.M  fully  expects  this  to  be  the  case. 


•  The  wirefrajne  data  correlates  with  edges  in  the  digitized  imagery  at  a  level  sufficient  to 

meet  accuracy  requirements. 

Modifications  to  implement  hidden  line  removal  will  be  required  given  that  this  data 
is  obscured  in  the  reaJ  images  (e  g.,  there  are  no  edges  for  fins  on  the  far  side  of  the  mis¬ 
sile).  While  early  studies  support  optimism  for  correlation  of  edges  to  the  wireframe  model, 
improved  contrast  and  accuracy  may  be  required.  Utilization  of  imagery  obtained  through 
color  filters  matched  to  the  film  response  will  be  explored  as  appropriate  to  assess  the  impact 
on  system  accuracy. 

Digitized  imagery  from  the  manual  GADS  system  will  be  processed  for  automated  op¬ 
erations.  While  this  is  an  obvious  input,  it  is  mentioned  here  to  emphasize  the  need  for  a 
standard  set  of  images  by  which  the  automated  processing  will  be  evaluated.  It  is  recom¬ 
mended  that  personnel  at  Eglin'select  a  representative  set  of  filmed  drops  to  provide  a  basis 
for  evaluation  of  the  algorithm  results.  The  set  of  digital  images  needs  to  span  the  range  of 
imaging  conditions  over  which  the  algorithm  performance  will  be  evaluated. 

Operator  interactions  are  of  two  forms:  those  inputs  requested  by  the  automated  GADS 
system,  and  requests  initiated  by  the  operator.  The  minimum  algorithm  inputs  anticipated 
are  selection  of  the  first  image  frame,  the  target  model,  and  initial  placement  of  the  model 
within  the  first  frame.  This  operation  is  envisioned  as  identical  to  manual  GADS  operations 
with  the  additional  option  of  shifting  into  automatic  when  the  model  is  sufficiently  close 
to  the  target.  Given  a  "close  enough”  placement  of  the  model,  the  automatic  mode  would 
perform  the  closing  match  to  the  target.  After  initialization,  the  operator  always  has  the 
option  of  switching  between  manual  and  automatic  model  placement  at  any  time  and  on  any 
frame.  When  in  automatic  mode,  the  operator  may  optionally  approve  or  reject  the  final 
model  placement  before  the  location  results  are  logged. 


8.0  ALGORITHM  OUTPUTS 

Li  the  upgraded  GADS  system,  the  outputs  will  be  identical  whether  from  automatic  or 
manual  placement  of  the  model.  In  particular,  the  following  outputs  are  anticipated: 

•  Model  location  in  x.y.z.yaw. pitch. roll 

•  Covariance  matrix  from  error  analysis  (optional) 

•  Display  of  the  image  and  model  placement  (optional  in  automatic  mode) 

•  Various  alphanumeric  information  to  a  terminal  (same  display  for  manual  or  automatic) 

The  model  location  values  are  those  currently  in  use  by  manual  GADS.  The  covariance 
matrix  would  reflect  the  coupling  of  errors  between  the  various  degrees  of  freedom  and  could 
be  used  in  a  Kalman  filtering  process.  Calculation  of  this  matrix  will  also  reflect  the  strength 
of  image  features  (edges)  and  their  correlation  with  the  model  data.  Finally,  the  last  two 
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outputs  mentioned  emphasize  the  close  relationship  between  manucil  and  automatic  modes. 
.\s  previously  mentioned,  automatic  mode  is  simply  an  option  the  operator  invokes  for 
cancels)  when  running  the  GADS  program.  Notice  that  this  upgrade  optionally  provides  the 
operator  numerical  feedback  on  the  manual  placement  of  the  model  before  location  vcilues  axe 
logged.  Given  that  this  approach  is  essentially  an  extension  of  the  current  .nanual  operations, 
this  effort  has  been  locally  referred  to  as  the  Enhamced  Graphics  Attitude  Determination 
System  or.  alternatively,  the  Graphics  Attitude  Determination  System  Utilizing  Knowledge 
Enhancement  Services. 


9.0  ALGORITHM  CALCULATIONS 

From  the  distance  field  generate-,  for  the  model,  we  know  the  amount  to  shift  in  lines 
and  columns  to  align  a  given  feature  image  pixel  with  the  nearest  model  pixel.  These  shifts 
are  a  function  of  pixel  location  and  can  be  represented  as  a  transformation  between  the 
coordinates  of  the  feature  image  and  the  model  image.  Using  a  homogeneous  coordinate 
system  to  facilitate  matrix  representation  of  translation  and  perspective,  the  transformation 
was  represented  as  follows: 
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The  equations  for  determining  the  final  coefficients  use  the  value  1  in  place  of  1*.  i.e..  wo 
assume  either  translation  is  small,  perspective  change  is  small,  or  that:  fypo  ~  Gpi 

Symbol  definitions  for  this  set  of  equations  are  as  follows: 


H 


Transformed  j  new  i  image  column  location 


V  Transformed  (new)  image  line  location 

w  Scale  of  homogeneous  solution,  result  vector  is  divided  by  w  to  get  u,  v. 

X  Original  column  position  in  the  image 
y  Originai  line  location  in  the  image 
tz  Number  of  column  i  (pixels)  to  traxislate 
ty  Number  of  rows  (lines)  to  translate 
a  Change  in  scale,  i.e.,  Znewl^oid 

IP  The  angle  of  iv  the  plane  image  rotation,  counterclockwise  from  x 

7  The  angle  of  out  of  plane  image  rotation 

K  The  in  plane  angle  to  the  out  of  plane  rotational  axis,  counterclockwise 
from  u 

Cy^,  Syj  Respectively,  the  cosine  arid  sine  of  ip 

d  The  distance  from  the  focal  plane  to  the  image  piant  m  units  of  pixels 
pi,  p2  Initial  perspective  terms 

Pj,  P2  Final  tranform  perspective  terms 

The  coefficients  of  the  transformation  are  obtained  from  a  weighted  least  squares  solu¬ 
tion  to  the  large  set  of  simultaneous  equations  for  the  pixel- to- pixel  correspondences.  The 
weighting  allows  the  strength  of  the  feature  and  other  image  properties  to  influence  the 

solution.  With  this  approach,  we  are  solving  for  the  translation  of  image  features  to  the 

wireframe  model.  In  the  application,  this  transformation  must  be  inverted  to  provide  for  the 
manipulation  of  the  wireframe  model. 


We  solved  for  object  point*  model  points,  but  we  need  the  model  to  object  transformation 
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Gathering  terms  yields 
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For  small  translations  or  small  changes  in  perspective,  we  can  assume 
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Similaxly 


P'.o  =  — (Pi3^  +  P2CV-) 
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The  transformations  utilized  by  the  model  generation  progrtim  must  be  compensated  for 
when  calculating  input  to  the  wireframe  model.  In  addition,  the  model  parameter  inputs 
must  be  the  total  or  cumulative  position  rather  than  incremental  changes  obtained  from  the 
above  calculations.  The  transformations  used  in  model  generation  axe  as  follows: 
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Symbol  definitions  used  in  these  transformation  matrices  au-e  given  below. 

tx  Camera  x  translation,  in  h  es  from  model  origin 
ty  Camera  y  translation,  in  inches  from  model  origin 
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tt  Caxnera  z  translation,  in  inches  from  model  origin 
r  Model  roU  cingle,  Sr  =  sin(r),  Cj-  =  cos(r) 
p  Model  pitch  angle 

y  Model  yaw  angle 


Finally,  the  coefficients  obtained  from  the  field  calculations  were  related  to  the  inputs 
required  by  the  model  generation  program  by  the  following  approximations. 
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This  modeling  of  the  transformation  coefficients  proved  adequate  for  till  in  plane  model 
movement.  A  demonstration  of  this  was  shown  in  the  exjimple  sequence  where  translation 
(x.y),  scaling  (z),  cind  pitch  were  simultaneously  displaced  from  the  image  feature  model. 
Initial  closure  to  the  feature  position  was  very  rapid  in  all  four  degrees  of  freedom,  with  the 
asymptotic  approach  partially  explained  by  the  potential  field  characteristics.  The  remaining 
two  degrees  of  freedom  for  the  out  of  image  plane  treinsformations  were  not  correctly  modeled. 
While  this  prevented  coupling  with  the  model  generation  program,  it  did  not  prevent  the 
direct  transformation  of  the  wireframe  image.  The  results  of  some  tests  allowing  six  degrees 
of  freedom  are  shown  in  Figures  22  through  24.  Only  the  yaw  parameter  was  different 
between  the  wireframe  model  and  image  features  in  Figure  22.  As  the  blue  outline  indicates, 
a  reasonable  transformation  did  occur  for  six  degrees  of  freedom.  Figure  23  suggests  that 
simultaneous  closure  in  six  dimensions  may  be  possible  even  under  fairly  large  displacements. 
The  exception  to  this  is  shown  in  Figure  24,  where  model  placement  was  too  remote  to  meet 
the  proximity  assumptions  in  the  algorithm  Cciicuiations.  ERIM  anticipates  a  final  automatic 
algorithm  will  restrict  the  number  of  parameters  initially  varied  to  ensure  stable  closure. 
In  particular,  the  sequence  would  be  to  first  translate,  then  permit  scaling  and  in  plane 
rotations,  and  finally  allow  out  of  plane  angular  corrections  only  after  the  model  and  feature 
data  are  closely  aligned. 


10.0  SYSTEM  ACCURACY 

During  the  meetings  held  at  Eglin  .Air  Force  Base  on  September  10  and  11.  several  design 
considerations  were  discussed.  Of  particular  interest  to  an  error  analysis,  the  following  item 
was  noted  in  .Attachment  2,  Summary  of  Discussion. 

The  desired  system  accuracy  is  two  (2)  inches  in  object  space  at  a  distance 
of  50  feet  to  the  target  with  a  one  (1)  inch  focal  length  lens. 


In  order  to  responsibly  assess  the  feasibility  of  this  accuracy,  ERIM  initiated  a  small  scale 
effort  to  determine  the  impact  of  various  errors  on  system  accuracy.  Some  of  the  items 
investigated  are  given  below. 

•  .Accuracy  of  film  geometry,  i.e.,  size  and  position  of  the  sprocket  holes. 

•  Consistency  of  film  geometry,  i.e.,  the  amount  of  shrinkage  or  warping. 

•  Positional  accuracy  of  the  film  during  exposure. 

•  Positional  accuracy  of  the  film  copying  process. 

•  Positional  accuracy  of  the  film  transport  being  used  for  digitization/review 

•  Scanning  accuracy  of  the  digitization  process. 
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In  addition  to  these  items,  consideration  must  be  given  to  focal  plane  distortion  from  optics 
and  the  variability  of  digitization.  Several  of  these  errors  were  readily  available  from  the 
manufacturers.  Anaiysis  of  the  accuracy  values  greatly  influenced  the  selection  of  upgraded 
GADS  equipment.  The  basic  geometry  and  measurements  for  the  film  are  given  in  Figures 
25  and  26. 

Applying  the  same  accuracy  requirements  in  object  space  for  16mm  and  35mm  film 
presents  a  problem  since  the  algorithm  is  working  in  image  space  with  pixels.  Thus,  one 
inch  represents  4  pixels  for  35mm  film  and  10  pixels  for  16mm  film.  Either  of  these  pixel 
distances  may  appeeir  generous  until  other  system  conditions  are  considered.  Using  the  pixel 
sizes  given  in  Figures  25  and  26,  the  manufacturers’  published  film  tolerances  are  given  in 
the  table  below.  Dimensions  are  given  in  pixels,  8.62/im  per  pixel  for  16mm,  21.22/im  per 
pixel  for  35mm. 


Film 

Column 

Line 

Line  (over  100  perfs) 

I6mm 

3.48 

1.51 

92.81  (100  perfs  = 

100  frames) 

35  mm 

2.37 

2.45 

17.91  (100  perfs  = 

25  frames) 

The  tolerance  for  100  film  perfs  represents  over  18  inches  in  object  space  for  16mm  film, 
and  just  under  9  inches  in  object  space  for  25  frames  of  35mm  film.  For  analysis  performed 
with  film  copies  of  the  original,  the  minimum  increeise  in  tolerance  would  be  additive,  or 
a  doubling  of  these  figures.  .Add  to  this  any  positional  errors  from  transport  equipment 
(camera,  copying,  digitization)  including  adjustments  for  sprocket  and  mechanism  wear,  and 
an  overall  system  accuracy  assessment  becomes  very  difficult  to  quantify. 

For  the  purposes  of  assessing  an  automatic  matching  algorithm,  overall  accuracy  consid¬ 
erations  could  be  avoided  by  utilizing  a  standard  set  of  digital  imagery.  Given  that  the  true 
location  of  the  target  is  not  available,  assessing  automated  performance  probably  entails  a 
comparison  with  manual  operations.  One  possible  comparison  would  be  of  the  operator  and 
automated  residuals  from  the  fitted  trajectory  of  the  target.  The  automatic  algorithm  will 
minimize  the  aggregate  distance  of  the  extracted  image  features  to  the  model  data  to  obtain 
an  optimal  overall  fit.  ERIM  anticipates  this  procedure  will  result  in  residuals  with  a  smaller 
standard  deviation  and  perhaps  a  small  bias  when  compared  with  residuals  from  manual 
target  placement. 

.As  a  final  note  on  this  subject,  the  sensitivity  of  the  results  in  each  of  the  six  degrees  of 
freedom  needs  to  be  considered.  .As  an  example  of  this,  consider  a  wireframe  model  viewed 
directly  ‘‘head-on.”  Changes  in  roll  from  this  perspective  should  be  resolvable  to  small 
angles.  In  contrast,  small  changes  in  pitch  or  yaw  are  unlikely  to  result  in  a  change  in  the 
digitized  representation  of  the  model.  Rendition  errors  will  also  arise  in  the  potential  field 
and  when  considering  the  features  extracted  from  digital  imagery.  .An  analysis  of  errors  must 
be  able  to  accommodate  all  these  sources. 


13 


11.0  SUMMARY  AND  RECOMMENDATIONS 


An  algorithm  for  automatically  matching  wireframe  model  data  to  edges  extracted  from 
digitized  film  images  was  designed  and  the  critical  portions  were  developed  and  demon¬ 
strated.  The  approach  leverages  existing  hardware  amd  software,  interfaw;ing  with  am  oper¬ 
ator  via  integration  with  manuaJ  operations.  From  an  algorithmic  perspective,  ERIM  would 
expect  future  work  on  automation  of  GADS  model  matching  to  address  the  following  areas: 

•  Resolve  the  final  2-degrees  of  freedom 

o  Separate  parts  with  invariant  shapes  for  an  aixis  of  rotation 
0  Reformulate  equations  utilizing  8  coefficients 

•  Image  feature  extraction  strategies  (Algorithm  step  2,  edges) 

0  Color  based  imagery 

0  Edge  linking  or  removal  for  regions  of  interest 
o  Edge  thinning,  minimum  continuity  criteria 
0  Integration  with  factors  arising  from  potential  field  model 

•  Potential  field  generation 

0  Continuous  domain  to  discrete  domain  approximations 
0  Region  of  stability  (what’s  close  enough?) 

0  Correct  correlation  closure  characteristics  (rate,  damping) 

•  Transformation  determination  (least  squares) 

0  Sensitivity  to  extraneous  or  incorrect  image  features 
o  Treatment  of  aligned  versus  non-aligned  model  pixels 
o  Weighting  of  images  features 

•  Error  analysis 

o  Covariance  of  errors  (model  parameters) 

0  Quantization  errors  (edge  accuracy,  model  rendition,  field  rendition) 

0  Determine  system  accuracy  (film,  camera,  transport,  digitization,  . . .) 

While  several  challenging  areas  remain  for  development,  the  results  of  this  feasibility 
study  indicate  an  automated  upgrade  to  manual  GADS  is  achievable.  The  development  of 
this  capability  would  also  enhance  manual  efforts  for  model  matching  through  the  quanti¬ 
tative  feedback  of  model  fit  and  error  analysis  results.  Given  this  approach  to  automated 
GADS  is  possible  and  well  integrated  with  existing  capability,  further  development  appears 
appropriate  and  is  encouraged. 
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Appendix  A 

LITERATURE  SURVEY 
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ERJM  performed  a  literature  search  using  the  computer  based  Dialog  Information  Re¬ 
trieval  Service.  A  description  of  the  four  database  files  queried  is  given  below. 

NTIS  Produced  by  the  National  Technical  Information  Service  of  the  U.S.  De¬ 
partment  of  Commerce,  the  central  source  for  the  public  s«de  and  dissem¬ 
ination  of  U.S.  government-sponsored  research,  the  database  represents 
reports  from  the  U.S.  Department  of  Energy  (DOE),  U.S.  Department  of 
Defense  (DOD),  Nationad  Aeronautics  and  Spawre  Administration  (NASA), 
in  addition  to  several  other  agencies. 

INSPEC  The  Information  Services  for  Physics,  Electronics,  and  Computing  is  the 
largest  English-language  databtise  for  the  fields  of  physics,  electrical  engi¬ 
neering,  electronics,  computers,  control  engineering,  and  information  tech¬ 
nology.  It  scans  over  3,900  journals  and  serials,  and  includes  source  mate¬ 
rial  from  conference  proceedings  and  papers  (approximately  20%),  books, 
dissertations,  and  reports. 

SPIN  The  Searchable  Physics  Information  Notices  covers  all  major  areaa  of 
physics.  References  are  drawn  from  cdl  English-language  and  Russian 
journal  articles  and  conference  proceedings  published  by  the  AlP  and  its 
member  societies  along  with  other  select  American  journals. 

SUPERTECH  This  database  consists  of  five  subfiles  with  information  related  to  the  fields 
of  biotechnology,  artificial  intelligence,  CAD/CAM,  robotics,  and  telecom¬ 
munications.  References  are  from  over  9,000  worldwide  sources  including 
trade,  scientific,  legal,  and  policy  journals;  profession8d  and  technological 
conferences  and  symposia;  university  monographs;  U.S.  and  international 
government  agency  reports  and  white  papers,  and  independent  laboratory 
studies. 

Using  a  keyword  search,  the  following  number  of  titles  were  obtained  from  each  file  after 
initially  reducing  to  only  those  entries  that  matched  the  keyword  image()process?.  Note 
that  keyword  usage  of  “()”  prohibits  matches  with  intervening  words,  while  the  “?”  gives  a 
match  for  any  word  ending. 


Keyword 

NTIS 

INSPEC 

SPIN 

SUPERTECH 

registration 

159 

90 

3 

15 

feature()detect? 

7 

9 

1 

15 

image(  )sequence(  )analysis 

1 

6 

0 

1 

time()varying()imag? 

15 

23 

0 

9 

For  the  two  instances  where  more  than  25  entries  matched  both  keywords,  only  the  first  25 
were  printed  and  examined.  After  this  initial  query,  a  second  search  was  initiated  with  the 
following  keywords  and  results: 
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Keyword 

NTIS 

INSPEC 

image(  )sequence(  )^maly3is 

1 

17 

time()varying()imag? 

17 

85 

motion  &  image()sequence 

6 

96 

moving()objects  &  image()sequence 

3 

17 

Once  again,  only  the  first  25  entries  were  reviewed  for  matches  with  over  25  titles.  A  pan'ti2d 
list  of  titles  obtained  from  these  searches  and  other  efforts  is  given  below. 

“3-0  Motion  Estimation  from  Projections,”  SPIE  Applications  of  Digital  Image 
Processing  IX  (1986) 

Temporal  Edge- Based  Image  Segmentor,”  Pattern  Recognition,  Vol.  20,  No. 

3,  1987 

'‘.\daptive  Gray  Scale  Mapping  to  Reduce  Registration  Noise  in  Difference  Im¬ 
ages,”  Computer  Vision,  Graphics,  and  Image  Processing,  33  (1986) 

‘‘An  Adaptative  Image  Sequence  Filtering  Scheme  Based  on  Motion  Detection,” 
SPIE  Architectures  «ind  Algorithms  for  Digital  Image  Processing  (1985) 

“.‘Application  of  the  One- Dimensional  Fourier  Transform  for  Tracking  Moving 
Objects  in  Noisy  Environments,”  Computer  Vision.  Graphics,  and  Image  Pro¬ 
cessing  21  (1983) 

“Correlation  of  Adjacent  Pixels  for  Multiple  Image  Registration,”  IEEE  Transac¬ 
tions  on  Computers  (7/85) 

“Extracton  of  Motion  Data  by  Interactive  Image  Processing,”  IEEE  CH1318 
(5/78) 

“Detection  of  Moving  Objects  Using  Line  Image  Sequence,”  IEEE  CH2046  (1/84) 

“Dynamic  Imagery  Analysis  via  Distributed  Parameter  Systems:  Characteristics, 
Discontinuities,  Weak  Solutions  and  Shocks,”  IEEE  CH1761  (6/82) 

“Man-Made  Objects  Dynamic  Recognition  Using  Gray  Levels  Corners,”  SPIE 
Architectures  and  Algorithms  for  Digital  Image  Processing  (1985) 

“Motion  Modeling  and  Prediction,”  IEEE  CH2342  (4/86) 

“Parameter  Space  Techniques  for  Image  Registration,”  Proceedings  of  IGARSS 
1986  Symposium 

“Recursive  Image  Registration  with  Application  to  Motion  Estimation,”  IEEE 
Transactions  on  Acoustics,  Speech  and  Signal  Processing  (1/87) 

“Registration  and  Rectification  of  Images  Using  Image  Algebra,”  IEEE  CH2398 
(6/87) 
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“Residual  Recursive  Displcicement  Estimation,”  IEEE  CH1595  (8/81) 

“Robust  Estimation  of  Three-Dimensional  Motion  Pcirameters  from  a  Sequence 
of  Image  Frames  Using  Regularization,”  IEEE  PAMI  (7/86) 

“Three-Dimensional  Motion  Analysis  using  Shape  Change  Information,”  SPIE 
Applications  of  Artificial  Intelligence  III  (1986) 

“Unique  Recovery  of  Motion  and  Optic  Flow  via  Lie  Algebras” 

“Visual  Disparity  as  a  Cue  for  Depth  and  Velocity  in  Read  World  Scenes,”  IEEE 

CH1318  (5/78) 

Additional  literature  from  various  manufacturers  also  proved  helpful  when  studying  system 
accuracy  issues.  ERIM  will  be  pleaised  to  provide  copies  of  these  papers  as  requested  by  Eglin 
personnel. 
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FIGURES 
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Potential  Field  Calculation 
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Figure  2:  Potential  Field  Perspective  Plot 
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Figure  4:  Potential  Field  Vertical  Component 


Figure  5.  Example  Image  Frame 


Figure  6.  Image  Feature  Determination 


Figure  7.  Image  Feature  Extract 
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Figure  9.  Initial  Model  Matching  Conditions 
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Figure  10.  Horizontal  Field  for  Wireframe 
Model 


Figure  11.  Vertical  Field  for  Wireframe  Model 
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Figure  13.  Loop  1,  Idealized  Features  in 
Vertical  Field 
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Figure  14.  Loop  1,  Model  Relocation 


Figure  15.  Loop  20,  Idealized  Features  in 
Horizontal  Field 


Figure  16.  Loop  20,  Idealized  Features  in 
Vertical  Field 
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Figure  17.  Loop  20,  Model  Relocation 
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Model  Z  Parameter 
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Figure  20:  Closure  of  Model  to  Features,  Z  Parameter 


Model  I’ilch  Parameter 
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Figure  21:  Closure  of  Model  to  Features,  Pitch  Parameter 
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Figure  23.  Perturbation  in  X,  Z,  Roll 
Pitch,  Yaw 


Figure  24.  Perturbation  Beyond  Stable 
Response 


EGLIN  GAOS  -  SYSTEM  ACCURACY 

GOAL:  2*  in  object  space  at  a  distance  of  50*  writh  a  1*  focal  length  lens 
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Figure  25:  Geometry  for  16mm  Film 


EGLIN  GAOS  SYSTEM  ACCURACY 

GOAL;  2*  in  object  space  at  a  distance  of  50*  with  a  1*  focal  length  lens 
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Figure  26:  Geometry  for  35mm  Film 


